sqlite: add virtual table support via createModule() - #65787
Open
TrevorBurnham wants to merge 1 commit into
Open
sqlite: add virtual table support via createModule()#65787TrevorBurnham wants to merge 1 commit into
TrevorBurnham wants to merge 1 commit into
Conversation
Collaborator
|
Review requested:
|
TrevorBurnham
force-pushed
the
sqlite-vtab-create-module
branch
2 times, most recently
from
September 4, 2026 19:28
7862ab0 to
69f939a
Compare
Expose SQLite's virtual table API through a new `database.createModule(name, options)` method, wrapping `sqlite3_create_module_v2()`. This enables read-only virtual tables backed by JavaScript data sources, usable either as an eponymous table (`SELECT * FROM module_name`) or via `CREATE VIRTUAL TABLE t USING module_name`. Hidden columns pass parameters using table-valued function syntax (`SELECT * FROM module_name(param1, param2)`). `options` accepts `columns`, `rows`, `directOnly`, and `useBigIntArguments`. Column types are validated against INTEGER, TEXT, REAL, BLOB, and ANY, and column names are quoted when building the `sqlite3_declare_vtab()` schema. Rebased from nodejs#61544, which was opened by byteforge38 and became inactive. Changes on top of that work: - xColumn reports the value each hidden column was constrained to, rather than NULL. SQLite treats xBestIndex's `omit` as a hint, so it may recheck a constraint it already handed to xFilter; against NULL that recheck rejected every row, and `gs(1, 3) WHERE start = 1` returned no rows. - xBestIndex lowers estimatedCost as it consumes constraints. With a constant cost the planner was free to pick the unconstrained plan and recheck afterwards, so a correlated parameter such as `FROM t, gs(t.a, t.a + 1)` also returned no rows. - Violations of the iteration protocol report a SQLite error instead of calling PropagateJSError with no JavaScript exception pending. That left `.all()` returning undefined and `exec()` reporting success. - xBestIndex passes the constrained hidden-column indices to xFilter through idxStr rather than an int bitmask, which previously aliased for parameter indices at or above the width of an int. - xFilter, xNext, and xColumn take a CallbackDepthGuard. Without it close() from inside rows(), an iterator's next(), or a row getter finalized the statement that SQLite was still stepping, crashing the process. - xClose calls the iterator's return() method so generator `finally` blocks run when SQLite stops stepping early, as it does for LIMIT or a `break` out of a for...of loop. It is skipped while tearing down from ~StatementSync or ~DatabaseSync, which run from garbage collection callbacks where JavaScript cannot be executed; an abandoned generator does not run `finally` in JavaScript either. It is also skipped when an error is already pending, so that error still reaches the caller. - VirtualTableModule holds a BaseObjectWeakPtr<DatabaseSync> to match UserDefinedFunction instead of a raw pointer. - createModule() rejects being called from an authorizer callback. - Documents that values yielded by rows() follow the usual conversion rules, so a number is stored as REAL and a BigInt as INTEGER even when a column declares INTEGER, since virtual tables do not apply column affinity to the values they return. Refs: nodejs#61544 Refs: nodejs#63826 Fixes: nodejs#61539 Co-authored-by: byteforge38 <[email protected]> Signed-off-by: Trevor Burnham <[email protected]> Assisted-by: Claude Opus 5
TrevorBurnham
force-pushed
the
sqlite-vtab-create-module
branch
from
September 4, 2026 21:04
69f939a to
89b3a3a
Compare
TrevorBurnham
marked this pull request as ready for review
September 4, 2026 23:17
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #65787 +/- ##
==========================================
- Coverage 90.13% 90.12% -0.02%
==========================================
Files 769 769
Lines 261645 262109 +464
Branches 49671 49755 +84
==========================================
+ Hits 235831 236217 +386
- Misses 16845 16856 +11
- Partials 8969 9036 +67
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #61539
This PR is a continuation of #61544 by @byteforge38, which became inactive.
It exposes SQLite's virtual table API through a new
database.createModule(name, options)method, wrappingsqlite3_create_module_v2().This PR enables read-only virtual tables backed by JavaScript data sources. A registered module can be used two ways:
SELECT * FROM module_name).CREATE VIRTUAL TABLE t USING module_name.Hidden columns pass parameters via table-valued function syntax (
SELECT * FROM module_name(param1, param2)).optionsacceptscolumns,rows,directOnly, anduseBigIntArguments.Note: column affinity
A virtual table doesn't apply column affinity to the values
xColumnreturns, which makes declared types behave differently than they do on an ordinary table:This follows from the conversion rules discussed in #63826: A
numberis bound asREAL, abigintasINTEGER. Value-based coercion would make it impossible to yield666.0into aREALcolumn, andbigintalready gives callers explicit control.Coercing to the declared type would be a different mechanism, driven by explicit intent rather than by guessing from the value, and it would leave
REALcolumns alone. But SQLite core doesn't do this for virtual tables (generate_seriesreturns integers because it callssqlite3_result_int64, not via affinity), and it adds per-cell cost.I've documented the current behavior rather than changing it, since coercing to the declared type is behavior SQLite core doesn't have for virtual tables.